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Next  Generation  Diagnostic  System  Increment  1 
Early  Fielding  Report 


Summary 

This  report  provides  the  Director,  Operational  Test  and  Evaluation’s  (DOT&E) 
operational  assessment  of  the  Next  Generation  Diagnostic  System  (NGDS)  in  support  of  a 
decision  to  field  it  to  Air  Eorce  clinical  laboratories  in  April  2017  prior  to  Initial  Operational  Test 
and  Evaluation  for  shipboard  operation  and  environmental  sample  analysis  capability.  The 
Army,  Navy,  and  Air  Eorce  intend  for  NGDS  to  replace  the  currently  fielded  Joint  Biological 
Agent  Identification  and  Diagnostic  System,  which  identifies  biological  warfare  agents  in 
clinical  and  environmental  samples,  to  increase  the  breadth  of  diagnostic  coverage  and  reduce 
the  manpower,  training,  and  logistics  burden  on  deployable  medical  units. 

This  evaluation  is  supported  by  data  from  the  operational  assessment  (OA)  of  land-based 
clinical  laboratory  use  of  the  NGDS  conducted  by  the  Army  Test  and  Evaluation  Command 
(ATEC),  in  coordination  with  the  Air  Eorce  Operational  Test  and  Evaluation  Command  and  the 
Navy’s  Commander,  Operational  Test  and  Evaluation  Eorce,  from  May  17  -  25,  2016,  at  Camp 
Bulbs,  Texas.  Data  from  developmental  testing  conducted  between  July  2015  and  December 
2016  and  the  U.S.  Eood  and  Drug  Administration  (EDA)  review  of  the  request  for  classification 
of  the  EilmArray  NGDS  Warrior  Panel  support  this  evaluation.  Test  and  evaluation  conducted  to 
date  is  adequate  to  assess  the  NGDS  operational  effectiveness,  operational  suitability,  and 
survivability  for  use  by  land-based  deployable  medical  units  in  support  of  medical  diagnostic 
decisions  for  symptomatic  patients. 

The  NGDS  is  operationally  effective  in  providing  land-based  deployable  medical  units 
with  timely  clinical  sample  analysis  to  aid  in  the  diagnosis  of  anthrax,  plague,  tularemia,  Q  fever, 
and  the  hemorrhagic  fevers  caused  by  Ebola  and  Marburg  viruses,  in  response  to  a  suspected  or 
confirmed  bioterrorism  event  or  outbreaks.  The  NGDS  provides  increased  breadth  of  diagnostic 
coverage  through  compatibility  with  four  EDA-approved  commercially  available  common 
infectious  diseases  panels  enabling  day-to-day  use  of  the  system.  Clinical  laboratory  personnel 
were  able  to  prepare  and  analyze  a  clinical  sample  in  an  average  of  72  minutes  versus  a 
requirement  of  90  minutes  and  correctly  report  diagnostic  results  for  multiple  agents  at  the  same 
time  to  support  medical  treatment  and  force  health  protection  decisions.  The  NGDS  automated 
sample  preparation  and  analysis  process  reduces  operator  sample  preparation  tasks  and 
minimizes  the  opportunity  for  error. 

The  NGDS  is  operationally  suitable  for  land-based  clinical  diagnostic  use.  The  NGDS  is 
easy  to  use.  The  infectious  disease  diagnostic  capability  enables  laboratory  personnel  to 
maintain  their  skillset  should  a  biological  warfare  agent  (BWA)  incident  occur.  The  NGDS 
demonstrated  98  percent  mission  reliability  over  56  analytical  runs,  exceeding  the  operational 
requirement  of  94.4  percent  probability  of  completing  5  analytical  runs  without  an  operational 
mission  failure.  The  NGDS  demonstrated  96.8  percent  operational  availability  during  the 
operational  assessment  exceeding  the  operational  requirement  of  85  percent. 
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The  classified  annex  to  this  report  identifies  cybersecurity  vulnerabilities  and  potential 
operational  impacts  that  could  occur  from  their  exploitation. 

A  complete  list  of  recommendations  appears  at  the  end  of  this  report.  Prior  to  full 
fielding  of  the  system,  DOT&E  recommends  the  following  to  improve  the  NGDS  cybersecurity 
posture: 

•  The  NGDS  Program  Manager  should  address  the  cybersecurity  vulnerabilities 
identified  in  the  classified  annex. 

•  The  NGDS  Program  Manager  should  conduct  cybersecurity  testing  on  the  NGDS 
production  representative  configuration,  which  includes  a  different  laptop  computer. 

System  Overview 

The  Services  intend  for  the  NGDS  to  replace  the  Joint  Biological  Agent  Identification 
and  Diagnostic  System  (JBAIDS)  to  identify  BWAs  in  clinical  and  environmental  samples  and 
infectious  diseases  in  clinical  samples  to  support  medical  treatment  decisions,  a  commander’s 
situational  awareness,  and  force  protection  decisions.  The  NGDS  is  a  commercial  off-the-shelf 
polymerase  chain  reaction  analytical  instrument  manufactured  by  BioFire  Defense,  Limited 
Liability  Corporation.  Polymerase  chain  reaction  technology  is  a  scientific  tool  used  to  detect 
and  identify  agent-specific  nucleic  acids  in  biological  samples.'  The  NGDS  program  contracted 
with  BioFire  to  develop  two  assay  panels:  the  Warrior  panel  to  identify  BWAs  in  clinical 
samples  and  the  Sentinel  panel  to  identify  BWAs  in  environmental  samples,  which  is  still 
undergoing  development  and  testing. 

Mission  Description  and  Concept  of  Employment 

The  Army,  Navy,  and  Air  Force  plan  to  employ  the  NGDS  Increment  I  in  medical 
support  facilities  with  existing  microbiology  laboratories  equipped  with  a  Class  II  Biosafety 
Cabinet,  refrigerator,  line  power  sources,  lighting,  and  appropriately  trained  personnel.  Trained 
clinical  laboratory  personnel  will  respond  to  medical  provider  requests  to  analyze  clinical 
samples  for  the  presence  of  BWA  or  infectious  diseases.  The  Services  intend  to  employ  NGDS 
in  existing  environmental  laboratories  in  the  future  to  analyze  environmental  samples  for  the 
presence  of  BWAs  and  to  inform  operational  and  medical  response  decisions. 

Figure  1  below  illustrates  the  process  for  clinical  use  of  the  NGDS. 


'  Nucleic  acids  include  deoxyribonucleic  acid  (DNA)  and  ribonucleic  acid  (RNA)  and  comprise  the  genetic 
material  of  bacteria  and  vimses. 
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CBRN  -  Chemical,  Biological,  Radiological,  Nuclear;  COA  -  Course  of  Action;  Cdr-  Commander; 
Prev  Med  -  Preventative  Medicine 

Figure  1.  NGDS  Clinical  Sample  Flow 


System  Description 

The  NGDS  consists  of  the  BioFire  FilmArray  2.0  Polymerase  Chain  Reaction  instrument, 
a  ruggedized  laptop  computer,  software,  a  ruggedized  transport  case,  a  pouch  loading  system  for 
sample  preparation,  an  optical  handheld  barcode  scanner,  an  optical  mouse,  power  and 
communications  cables,  commercial  off-the-shelf  and  government  off-the-shelf  consumable 
assays,  reagents,  and  protocols.  Figures  2  and  3  show  the  NGDS  hardware  and  consumables. 
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Figure  2.  NGDS  Analyzer,  Barcode  Scanner,  Laptop,  and  Sample  Preparation  Equipment 
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Figure  3.  NGDS  assay  panel  showing  internal  nucleic  acid  extraction,  polymerase  chain  reaction  chambers, 
and  multi-well  array  chamhers  used  during  sample  analysis. 
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Table  1  shows  the  four  FDA-approved  eommereially  available  panels  for  diagnosis  of 
oommon  infeetious  diseases  in  elinieally  relevant  matriees  available  for  purehase  and  use  with 
the  NGDS  instrument.  Tables  2  and  3  list  the  targets  in  the  Warrior  and  Sentinel  panels. 


Table  1,  NGDS  FDA-approved  Commercial  Off-the-Shelf  Assays 


Respiratory  Panel 

Blood  Culture  Identification  Panel 

VIRUSES 

GRAM-POSITIVE  BACTERIA 

Adenovirus 

Enterococcus 

Coronavirus  HKU1 

Listeria  monocytogenes 

Coronavirus  NL63 

Staphylococcus 

Coronavirus  229E 

Staphylococcus  aureus 

Coronavirus  OC43 

Streptococcus 

Human  Metapneumovirus 

Streptococcus  agalactiae 

Human  Rhinovirus/Enterovirus 

Streptococcus  pneumoniae 

Influenza  A 

Streptococcus  pyogenes 

Influenza  A/H1 

Influenza  A/H3 

GRAM-NEGATIVE  BACTERIA 

Influenza  A/H 1-2009 

Acinetobacter  baumannii 

Influenza  B 

Haemophilus  influenzae 

Parainfluenza  Virus  1 

Neisseria  meningitidis 

Parainfluenza  Virus  2 

Pseudomonas  aeruginosa 

Parainfluenza  Virus  3 

Enterobacteriaceae 

Parainfluenza  Virus  4 

Enterobacter  cloacae  complex 

Respiratory  Syncytial  Virus 

Escherichia  coli 

BACTERIA 

Klebsiella  oxytoca 

Klebsiella  pneumoniae 

Bordetella  pertussis 

Proteus 

Chlamydophila  pneumoniae 

Serratia  marcescens 

Mycoplasma  pneumoniae 

ANTIMICROBIAL  RESISTANCE  GENES 

mecA  -  methicillin  resistance 

vanA/B  -  vancomycin  resistance 

KPC  -  carbapenem  resistance 
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Table  1.  NGDS  FDA-approved  Commercial  Off-the-Shelf  Assays  (continued) 


Gastrointestinal  Panel 

Meningitis/Encephalitis  Panel 

BACTERIA 

BACTERIA 

Campylobacter  (jejuni,  coli  and  upsaliensis) 

Escherichia  coli  K1 

Clostridium  difficile  (toxin  A/B) 

Haemophilus  influenzae 

Plesiomonas  shigelloides 

Listeria  monocytogenes 

Salmonella 

Neisseria  meningitidis 

Yersinia  enterocolitica 

Streptococcus  agalactiae 

Vibrio  (parahaemolyticus,  vulnificus  and  cholerae) 

Streptococcus  pneumoniae 

Vibrio  cholera 

VIRUSES 

DIARRHEAGENIC  E.  COLI/SHIGELLA 

Cytomegalovirus  (CMV) 

Enteroaggregative  E.  coli  (EAEC) 

Enterovirus 

Enteropathogenic  E.  coli  (EPEC) 

Herpes  simplex  virus  1  (HSV-1) 

Enterotoxigenic  E.  coli  (ETEC)  It/st 

Herpes  simplex  virus  2  (HSV-2) 

Shiga-like  toxin-producing  E.  coli  (STEC)  stx1/stx2 

Human  herpesvirus  6  (HHV-6) 

E.  coli  0157 

Human  parechovirus 

Shigella/Enteroinvasive  E.  coli  (El EC) 

Varicella  zoster  virus  (VZV) 

PARASITES 

YEAST 

Cryptosporidium 

Cryptococcus  neoformans/gattii 

Cyclospora  cayetanensis 

Entamoeba  histolytica 

Giardia  lamblia 

VIRUSES 

Adenovirus  F40/41 

Astrovirus 

Norovirus  GI/GII 

Rotavirus  A 

Sapovirus  (1,  II,  IV  &  V) 

Table  2,  Warrior  Panel  Agents  and  Clinical  Sample  Matrices 


Disease 

Pathogen 

Clinical  Sample  Matrices 

Positive  Blood 
Culture 

Whole  Blood 

Anthrax 

Bacillus  anthracis 

X 

X 

Plague 

Yersinia  pestis 

X 

X 

Tularemia 

Francisella  tularensis 

X 

Q  Fever 

Coxiella  burnetii 

X 

Viral  Hemorrhagic 
Fever 

Ebola  Virus 

X 

Marburg  Virus 

X 
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Table  3,  Sentinel  Panel  Agents  and  Environmental  Sample  Matrices 


Biological  Warfare  Agent 

Environmental  Sample  Matrices 

Bacillus  anthracis 

Yersinia  pestis 

Francisella  tularensis 

Coxiella  burnetii 

Brucella 

Burkholderia 

Burkholderia 

Rickettsia 

EEE  Virus 

VEE  Virus 

WEE  Virus 

Smallpox 

Ebola  Virus 

Marburg  Virus 

Clostridium  botulinum 

Culture  medium 

Aerosol  collection  buffer  solution 

Powder/Surface  Water  collected  on  swabs 

Soil/Sand 

Animal  Blood 

Vectors  (such  as  insects) 

Test  Adequacy 

The  data  from  the  OA,  supplemented  by  data  from  developmental  testing  eondueted 
between  July  2015  and  Deeember  2016  and  the  FDA  review  of  the  FilmArray  NGDS  Warrior 
Panel,  was  adequate  to  support  this  evaluation. 

ATEC,  with  support  from  the  Army  Medieal  Department  Center  and  School  and  the  Air 
Force  Medical  Evaluation  Support  Activity,  conducted  the  OA  at  the  Deployable  Medical 
Systems  Equipment  Training  Site  at  Camp  Bulbs,  Texas,  from  May  17  -  25,  2016.  Army 
Medical  Department  Center  and  School  personnel  conducted  new  equipment  training  at  Eort 
Sam  Houston,  Texas,  from  May  9  -  12,  2016.  The  OA  was  conducted  in  accordance  with  the 
DOT&E-approved  test  plan. 

The  OA  consisted  of  15  military  personnel  from  the  Army,  Navy,  and  Air  Eorce,  staffing 
9  laboratories.  Each  laboratory  was  equipped  with  one  NGDS  system.  The  laboratories 
received,  prepared,  and  analyzed  502  threat  representative  clinical  samples  spiked  with  nucleic 
acids  from  routine  pathogens  and  BWAs  with  enough  DNA  to  elicit  a  response  from  the  system. 
A  laboratory  officer  oversaw  operations  at  each  of  the  laboratories  to  ensure  accuracy  and 
provide  guidance. 

Table  4  lists  the  OA  clinical  samples  provided  to  each  laboratory  during  the  test. 


7 


Table  4,  NGDS  Panel  and  Sample  Specimen  Provided  to  the  Laboratories  During  the  OA 


Threat 

Specimen 

Test  Panel 

Each  Lab 

Totals  per 
Specimen  Matrix 

Totals  for 
Microorganism 

Respiratory 

Nasal  Wash  + 

Respiratory 

4 

4 

17 

Nasal  Wash  - 

Respiratory 

13 

13 

Bacterial 

Blood  Culture  + 

Blood  culture 

3 

3 

7 

Blood  Culture  - 

Blood  culture 

4 

4 

Bacillus  anthracis 

Blood  Culture  + 

WP 

2 

2 

5 

Blood  Culture  - 

WP 

0 

Whole  Blood  + 

WP 

2 

3 

Whole  Blood  - 

WP 

1 

Coxiella  burnetii 
(Q  Fever) 

Whole  Blood  + 

WP 

3 

5 

5 

Whole  Blood  - 

WP 

2 

Francisella  tularensis 

Whole  Blood  + 

WP 

2 

2 

7 

Whole  Blood  - 

WP 

0 

Sputum  + 

WP 

4 

5 

Sputum  - 

WP 

1 

Yersinia  pestis  (bubonic) 

Blood  Culture  + 

WP 

2 

3 

6 

Blood  Culture  - 

WP 

1 

Whole  Blood  + 

WP 

3 

3 

Whole  Blood  - 

WP 

0 

Yersinia  pestis 
(pneumonic) 

Whole  Blood  + 

WP 

2 

2 

6 

Whole  Blood  - 

WP 

0 

Sputum  + 

WP 

2 

4 

Sputum  - 

WP 

2 

Marburg  Virus 

Whole  Blood  + 

WP 

2 

3 

3 

Whole  Blood  - 

WP 

1 

At  each  lab 

56 

56 

56 

Across  all  9  labs 

504 

WP  -  Warrior  Panel 
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The  OA  consisted  of  five  threat  representative  vignettes.  Each  vignette  had  notional 
Forward  Operating  Bases  where  personnel  contracted  an  illness  associated  with  a  BWA.  A 
Physician’s  Assistant  saw  notional  patients  from  each  of  the  Forward  Operating  Bases.  The 
patients  came  to  the  clinic  and  described  symptoms  consistent  with  both  BWA  and  routine 
pathogens.  The  Physician’s  Assistant  ordered  laboratory  testing  typically  available  within  Army 
Combat  Support  Hospitals  based  on  patient  signs  and  symptoms.  Each  laboratory  received 
samples  for  each  matrix  and  agent  on  the  NGDS  Warrior  Panel;  the  laboratories  received  the 
same  sample  at  the  same  time.  Consistent  with  realistic  operations,  the  test  team  provided  both 
positive  and  negative  samples  to  the  laboratories  and  one  set  of  results  was  provided  to  the 
Physician’s  Assistant.  The  patient’s  symptoms  evolved  over  time,  based  on  documented  disease 
progression.  The  Physician’s  Assistant  ordered  treatment  based  on  laboratory  results  from  the 
NGDS  and  notional  results  provided  by  the  test  team.  The  Physician’s  Assistant  made  force 
health  protection  recommendations  to  notional  commanders  based  on  the  vignette. 

The  Army  Research  Faboratory  -  Survivability  Fethality  Analysis  Directorate  conducted 
a  Cooperative  Vulnerability  and  Penetration  Assessment  from  April  19  -  21,  2016,  at  the  Army 
Medical  Department  Center  and  School  in  Fort  Sam  Houston,  Texas.  The  Army’s  Threat 
Systems  Management  Office  conducted  an  Adversarial  Assessment  May  23  -25,  2016,  during 
the  last  three  days  of  the  OA  Record  Test.  Both  assessments  were  conducted  in  insider  and 
nearsider  postures  using  the  standalone  NGDS  operational  configuration.^  Once  the  Services 
identify  NGDS  networking  requirements  for  laboratory  managements  systems  and  identify  plans 
to  integrate  the  NGDS  into  a  networked  configuration,  additional  cybersecurity  testing  will  be 
required. 

Subsequent  to  the  OA,  BioFire  Defense  decided  to  upgrade  the  NGDS  laptop  in  order  to 
utilize  the  Windows  10  operating  system  required  by  the  Department  of  Defense.  Additional 
developmental  and  cybersecurity  testing  are  required  to  verily  that  the  upgraded  laptop  and 
operating  system  have  not  had  a  negative  impact  on  performance  and  cybersecurity.  Operational 
and  cybersecurity  testing  with  the  new  NGDS  laptop  is  planned  for  May  -  June  2017. 

Test  Limitations 

The  OA  test  scenarios  may  have  affected  the  evaluation  of  the  time  to  diagnose  a  patient 
suffering  from  BWA  exposure.  The  OA  notional  index  patients  arrived  at  the  medical  clinic 
with  advanced  signs  and  symptoms  of  disease.  It  is  likely  that  military  personnel  would  seek 
medical  attention  before  their  symptoms  reached  this  stage  of  severity. 

Operational  Effectiveness 

The  NGDS  is  operationally  effective  in  providing  deployable  medical  units  timely 
clinical  sample  analysis  to  aid  in  the  diagnosis  of  anthrax,  plague,  tularemia,  Q  fever,  and  the 


^  An  insider  would  be  someone  within  the  organization  that  likely  has  permission  to  aeeess  the  system;  a 
nearsider  has  aeeess  to  the  spaee  but  would  not  have  permission  to  aeeess  to  the  system. 
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hemorrhagic  fevers  caused  by  Ebola  and  Marburg  viruses,  in  response  to  a  suspected  or 
confirmed  bioterrorism  event  or  outbreaks.  Proper  and  timely  diagnosis  supports  medical 
treatment  decision-making  and  force  health  protection  decisions. 

On  February  14,  2017,  the  FDA  authorized  BioFire  Defense  to  market  the  FilmArray 
NGDS  Warrior  Panel  for  use  with  the  FilmArray  2.0  system  as  a  medical  diagnostic  device  for 
use  by  Department  of  Defense  laboratories,  and  laboratories  designated  by  the  Department  of 
Defense.  The  FilmArray  NGDS  Warrior  Panel  detects  and  identifies  Bacillus  anthracis.  Yersinia 
pestis,  Francisella  tularensis,  Coxiella  burnetii,  Ebola  virus,  and  Marburg  virus  nucleic  acids 
directly  from  human  whole  blood.  Operators  will  use  the  FilmArray  NGDS  Warrior  Panel  to  test 
for  Bacillus  anthracis  or  Yersinia  pestis  nucleic  acids  in  blood  cultures.  A  copy  of  the  FDA 
letter  dated  February  14,  2017,  is  attached. 

Table  5  summarizes  the  NGDS  with  respect  to  operational  performance  requirements. 


Table  5,  Effectiveness 


Requirement 

Threshold 

Demonstrated 

Performance 

Force  Protection  -  Air  Force 

Bacillus  anthracis 
Hemorrhagic  Fever  Virus 
[Ebola  or  Marburg] 

Bacillus  anthracis 

Ebola  virus 

Marburg  virus 
Francisella  tularensis 
Yersinia  pestis 

Coxiella  burnetii 

Force  Protection  -  Army,  Navy 

Bacillus  anthracis 
Hemorrhagic  Fever  Virus 
[Ebola  or  Marburg] 
Francisella  tularensis 
Yersinia  pestis 

Coxiella  burnetii 

Regulatory  Compliance 

FDA-cleared  for  in  vitro 
diagnostic  use 

FDA-cleared  for  in  vitro 
diagnostic  use 

Sample  Preparation  Time  and 
Instrument  Time  to  Results 

90  minutes 

71.6  minutes 

Samples  per  Day 

16 

20 

FDA  -  Food  and  Drug  Administration 


Mission  Accomplishment 

Army,  Navy,  and  Air  Force  test  units  equipped  with  NGDS  demonstrated  the  capability 
to  use  the  system  to  analyze  clinical  samples  and  provide  accurate  and  timely  information  to 
medical  practitioners  and  command  staff  to  support  diagnostic  and  force  health  protection 
decisions.  The  NGDS  capability  to  analyze  environmental  samples  is  still  being  developed  and 
tested. 

The  NGDS  correctly  identified  the  target  in  all  of  the  samples  during  the  operational  test. 
During  the  test,  9  Service  medical  laboratories  used  the  NGDS  to  analyze  502  clinical  samples. 
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which  included  blank  samples  and  samples  spiked  with  synthetic  DNA.^  Table  6  shows  the 
NGDS  results  obtained  during  the  OA. 


Table  6,  NGDS  OA  Sample  Analysis  Results 


Target 

Runs 

Successfully 

Completed 

Number  of 
Samples  Reported 
Correctly 

Accuracy*  (80% 
Confidence  Interval) 

Bacillus  anthracis 

36 

36 

(0.94,  1] 

Coxiella  burnetii 

30 

30 

(0.93,  1] 

Francisella  tularensis 

54 

54 

(0.96,  1] 

Yersinia  pestis 

71 

71 

(0.97,  1] 

Marburg  Virus 

18 

18 

(0.88,  1] 

Strep  throat 

36 

36 

(0.94,  1] 

Flu 

36 

36 

(0.94,  1] 

Negative 

221 

221 

(0.99,  1] 

Total 

502 

502 

‘Calculated  the  80  percent  lower  confidence  interval  using  the  exact  Clopper-Pearson  method 


DOT&E  evaluated  the  timeliness  of  NGDS  diagnostic  results  to  support  force  health 
protection  decisions  based  on  the  operational  vignettes  and  disease  progression  timelines. 
During  the  test,  the  medieal  provider  ordered  initial  treatment  based  on  a  patient’s  signs  and 
symptoms  while  waiting  for  NGDS  or  other  laboratory  results.  The  medical  provider  used 
NGDS  diagnostic  results  to  tailor  antibiotic  treatment  to  the  specific  biological  agent  identified 
or  to  cease  medication  based  on  NGDS  results.  Based  on  NGDS  results  identifying  a  biological 
agent,  the  medical  provider  also  made  foree  health  protection  recommendations  to  command 
personnel.  Table  7  shows  the  overall  diagnosis  timelines  and  results  for  the  five  OA  vignettes. 
The  time  to  diagnose  a  BWA-related  disease  is  dependent  upon  how  quickly  sick  patients  seek 
medical  attention  and  when  a  medical  provider  orders  testing  for  a  BWA  threat. 


^  While  504  samples  were  planned,  during  the  OA  some  tests  had  to  be  re-run  because  of  internal  assay  control 
failures  or  NGDS  software  anomalies.  In  total,  502  samples  were  used  to  evaluate  NGDS  detection 
performance  and  505  were  used  to  evaluate  the  reliability  requirement. 
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Table  7.  Diagnostic  Timelines  for  Vignettes  Portrayed  During  NGDS  OA 


Vignette  - 
Agent 

Number  of 
Patients  with 
Disease 

Correct 

Diagnosis 

Time  to  diagnose 
BWA? 

Force  Protection 
Measures 

FOB  1- 
Bacillus 
anthracis 

4  -  Anthrax 

1  -  Flu 

1  -  Staph  infection 

1  -  Parainfluenza 

Yes 

4  days  post-exposure 

26  hours  after  patient 
arrived  at  clinic 

Isolate  Anthrax  patients 
Recommend  prophylactics  for 
soldiers  in  FOB  1 

FOB2- 

Coxiella 

burnetii 

3  -  Q-fever 

1  -  Pneumonia 

1  -  Asthma 

Yes 

Approximately,  4  days 
post-exposure 

27  hours  after  first 
patient  arrived  at  clinic 

Isolation  of  Q-fever  patients 

Notified  chain  of  command 

Tailored  antibiotic  regimen 

FOB  3- 

Franciseiia 

tuiarensis 

6  -  Tularemia 

1  -  Strep  throat 

1  -  Pneumonia 

Yes 

Approximately,  6  days 
post-exposure 

30  hours  after  first 
patient  arrived  at  clinic 

Tailored  antibiotics  regimen 

Notified  chain  of  command 

Recommended  prophylactic 
for  soldiers  in  FOB  3 

FOB  4- 

Yersinia  pestis 

9  -  Plague 

1  -  Staph  infection 

1  -  Flu 

Yes 

Approximately,  5  days 
post-exposure 

4  hours  after  first 
patient  arrived  at  clinic 

Notified  chain  of  command 

Recommended  pest  control 
spray  for  fleas 

Recommended  prophylactic 
for  soldiers  in  FOB  4 

Anyone  with  a  fever  >  1 00.4 
ordered  to  report  to  sick  call 
for  a  NGDS  Warrior  Panel  test 

FOB5- 
Marburg  Virus 

2  -  Marburg 

1  -  Staph  infection 

1  -  Flu 

Yes 

5  days  post-exposure 

70  minutes  after  first 
patient  arrived  at  clinic 
with  obvious 
hemorrhagic  fever 
symptoms 

Isolated  Marburg  patients 

Ordered  full  Personal 

Protective  Equipment  for  staff 

Notified  chain  of  command 

Ordered  screening  for  patients 
prior  to  entry  into  hospital 

FOB  -  Forward  Operating  Base;  BWA  -  Biological  Warfare  Agent 


The  time  to  reeeive  NGDS  test  results  and  diagnose  a  disease  was  dependent  upon  the 
sample  type  ordered  by  the  Physieian’s  Assistant.  Blood  eulture  samples  required  the  longest 
time  beeause  of  the  24-hour  eulture  time  required  prior  to  the  NGDS  analysis.  The  Centers  for 
Disease  Control  reeommends  blood  eultures  as  the  gold-standard  diagnostie  method  for  some 
baeterial  agents  sueh  as  Bacillus  anthracis  and  Francisella  tularensis.  For  blood  eulture 
samples,  blood  is  drawn  from  a  patient,  inoeulated  into  growth  media,  and  allowed  to  incubate 
for  at  least  24  hours  before  analysis.  For  whole  blood  and  sputum  samples,  the  time  to  diagnose 
a  disease  was  driven  by  the  time  the  NGDS  requires  to  prepare  and  analyze  a  sample. 

For  most  notional  patients,  medical  treatment  began  immediately  upon  entry  into  sick  call 
based  on  patient  signs  and  symptoms.  Treatment  included  broad-spectrum  antibiotics  and 
breathing  treatments.  The  Physician’s  Assistant  used  the  NGDS  results  to  refine  the  selection  of 
prescribed  medications,  tailoring  the  antibiotics  to  the  specific  agent  identified  or  ceasing 
medications  if  they  were  not  warranted  based  on  NGDS  results.  When  the  NGDS  Warrior  panel 
identified  BWAs,  the  Physician’s  Assistant  used  the  results  to  make  force  health  protection 
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recommendations  to  the  notional  commander,  including  isolation  and  quarantine  of  sick 
individuals,  antibiotic  prophylaxis,  and  monitoring  for  the  development  of  fevers  among  the 
affected  units. 

Operational  System  Performance 

The  NGDS  provides  significant  improvements  over  the  currently  fielded  Joint  Biological 
Agent  Identification  and  Diagnostic  System  (JBAIDS)  that  it  will  replace.  The  NGDS  supports 
diagnosis  of  BWA-related  illnesses  and  infectious  diseases  with  similar  initial  symptoms,  which 
may  contribute  to  more  rapid  recognition  of  a  BWA-related  event.  Table  8  compares  the  NGDS 
to  the  JBAIDS. 


Table  8,  JBAIDS  and  NGDS  Comparison 


System  Characteristic 

JBAIDS 

NGDS 

Sample  Preparation  Time 

20-40  minutes  per  sample 
(Platinum  Path  extraction  kit) 

3.8  minutes  per  sample 
[3.7,  3.9]  80%  Cl 

Instrument  Time  to  Result 

60-80  minutes 

67.8  minutes 
[67.7-68.0]  80%  Cl 

Samples  analyzed  per  run 

32  samples  per  run  (including 
controls) 

1  sample 

Number  of  organisms  analyzed 
per  run 

Max  10  targets  -  Operator 
must  know  what  to  test  for 

14-27  simultaneous  targets 
depending  on  the  specific  panel 

Types  of  organisms  analyzed 

Biological  Warfare  Agents, 
Influenza 

Biological  Warfare  Agents,  many 
common  infectious  disease 
agents  identified  in  Figure  3 

Cl  -  Confidence  Interval 


Operational  Suitability 

The  NGDS  is  operationally  suitable  to  support  medical  diagnostic  missions.  It  reduces 
the  manpower  burden  on  a  deployable  medical  unit  because  it  requires  less  operator  hands-on 
time,  is  less  complex  thereby  requiring  less  operator  training,  and  can  be  used  for  clinical 
diagnosis  of  a  range  of  infectious  diseases  to  maintain  operator  proficiency. 

Table  9  summarizes  the  NGDS  performance  with  respect  to  key  operational 
requirements. 


Table  9,  Suitability 


Requirement 

Threshold 

Performance 

Reliability 

(Probability  of  completing  5  runs 
without  an  operational  mission  failure) 

94.4% 

98.4%  (80%  LCB) 

Operational  Availability 

85% 

96.  8%  (80%  LCB) 

Health  Data  Output  Compliance 

Health  Level  7 

Health  Level  7 

LCB  -  Lower  Confidence  Bound 
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Reliability,  Availability,  and  Maintainability 

The  NGDS  met  the  user-defined  reliability  requirement  of  94.4  pereent  probability  of 
completing  5  runs  without  an  operational  mission  failure  with  80  percent  confidence  during  the 
OA.  It  did  not  experience  an  operational  mission  failure  during  502  analytical  runs.  The  NGDS 
experienced  seven  failures  during  the  OA  that  did  not  result  in  an  operational  mission  failure. 
Table  10  summarizes  the  NGDS  failures  that  occurred  during  the  OA  based  upon  their  effect  on 
the  mission. 


Table  10.  NGDS  Failures 


Failure  Type 

Number  of 
Failures 

Failure  Modes 

Mean  Runs 
Between  Failure 
(Point  Estimate) 

Mission 

Reliability  (80% 
LCB) 

Operational 
Mission  Failure 

0 

N/A 

N/A 

98.4% 

Mission  Affecting 
Failure 

3 

2  Control  failures 

1  analyzer  reboot 
required 

168 

97.0% 

Non-Essential 
Function  Failure 

2 

2  barcode  scanners 
inoperable 

251 

98.0% 

Rapid 

Recoverable 

Event 

2 

1  analyzer  reboot 

1  software  anomaly 

251 

98.0% 

LCB  -  Lower  Confidence  Bound 


Mission  Affecting  Failures  occurred  during  the  clinical  sample  analysis  because  of  assay 
internal  control  failures  or  an  analyzer  reboot  required  in  the  middle  of  a  run.  Assay  internal 
controls  ensure  the  chemistry  inside  each  assay  pouch  has  functioned  correctly.  When  a  control 
fails,  the  system  does  not  alert  the  operator  until  the  end  of  the  67-minute  run.  The  sample  must 
be  prepared  again  and  analyzed  a  second  time,  resulting  in  a  delay  of  approximately  70  minutes 
in  reporting  the  results.  The  Non-Essential  Function  Failures  occurred  due  to  an  inoperable 
barcode  scanner.  There  is  no  operational  impact  to  an  inoperable  barcode  scanner  because  the 
user  can  simply  type  the  barcode  into  the  NGDS  computer  software.  Operators  were  able  to 
quickly  recover  from  the  Rapidly  Recoverable  Event  failures  by  rebooting  the  analyzer  or  laptop 
without  having  to  prepare  and  run  another  sample. 

Developmental  testing  in  December  2016  identified  a  potential  NGDS  reliability 
problem.  After  exposure  to  operationally  realistic  transport  vibrations,  some  of  the  NGDS 
analyzer  cables  were  no  longer  connected.  The  manufacturer  has  agreed  to  make  design  changes 
to  address  the  problem.  Once  planned  design  changes  are  complete,  the  program  should  conduct 
a  verification  of  fix  test. 

The  NGDS  operational  availability  requirement  is  85  percent.  The  NGDS  demonstrated 
96.8  percent  availability  at  the  80  percent  confidence  level. 
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The  NGDS  vendor  is  responsible  for  maintaining  eonfiguration  eontrol  as  an 
FDA-approved  medieal  deviee.  As  sueh,  there  are  limited  operator-level  maintenanee  tasks  that 
can  be  performed  on  the  NGDS.  These  maintenance  tasks  involve  rebooting  the  analyzer  or 
laptop.  If  these  maintenance  tasks  do  not  resolve  the  problem,  operators  must  return  the  NGDS 
hardware  to  the  vendor  for  repair.  In  order  to  maintain  capability  while  systems  are  with  the 
vendor  for  repair,  the  Services  plan  to  maintain  spares  at  depots  and  other  pre-positioned  sites  to 
deploy  systems  as  needed. 

Training 

The  NGDS  reduces  the  manpower  burden  on  a  unit  supporting  clinical  diagnosis  because 
it  requires  less  operator  hands-on  time,  is  less  complex  thereby  requiring  less  operator  training, 
and  can  be  used  for  clinical  diagnosis  of  a  range  of  infectious  diseases  to  maintain  operator 
proficiency.  In  response  to  an  OA  survey,  NGDS  operators  indicated  that  training  provided  was 
adequate  to  support  operation  of  the  NGDS.  NGDS  operators  recommended  adjustments  be 
made  to  improve  the  flow  of  training  materials.  They  recommended  sample  preparation 
procedures  be  provided  in  one  laminated  set  of  procedures  for  use  during  sample  preparation. 

The  Physician’s  Assistant  recommended  that  medical  providers  at  receiving  units  be  provided 
training  on  the  NGDS  capabilities. 

Logistics 

The  NGDS  has  a  smaller  footprint  than  the  JBAIDS.  The  footprint  includes  the  analyzer, 
system  specific  support  equipment,  and  consumables.  The  JBAIDS  logistical  footprint  is 
approximately  1150  pounds  and  87  cubic  feet.  The  NGDS  logistical  footprint  is  approximately 
62  pounds  and  6  cubic  feet. 

Survivability 

The  evaluation  of  cybersecurity  vulnerabilities  and  the  potential  for  operational  impacts 
should  these  vulnerabilities  be  exploited  can  be  found  in  the  classified  annex  to  this  report. 

Subsequent  to  the  OA,  BioFire  Defense  upgraded  the  NGDS  laptop  in  order  to  implement 
the  Windows  10  operating  system  required  by  the  Department  of  Defense.  Additional 
cybersecurity  testing  on  the  production  representative  configuration  is  required  to  reassess  the 
cybersecurity  vulnerabilities  and  potential  operational  impacts  that  could  occur  from  their 
exploitation. 

Recommendations 

Prior  to  full  fielding  of  the  system,  DOT&E  recommends  the  following; 

•  To  realize  the  potential  of  the  NGDS  to  expedite  recognition  that  a  BWA  incident  has 
occurred,  the  Services  should  develop  and  implement  plans  to  educate  medical 
providers  at  units  receiving  NGDS  on  the  capabilities  provided  and  the  diversity  of 
assays  available  to  support  medical  diagnostics. 

•  To  improve  operational  suitability,  the  Joint  Program  Manager  for  Medical 
Countermeasures  should  revise  organization  of  training  materials  for  NGDS  operators 


15 


to  improve  the  logical  flow  of  information  and  provide  sample  preparation  procedures 
on  a  single  job  aid  instead  of  one  for  each  matrix  type. 

•  The  NGDS  Program  Manager  should  address  the  cybersecurity  vulnerabilities 
identified  in  the  classified  annex. 

•  The  NGDS  Program  Manager  should  conduct  cybersecurity  testing  on  the  NGDS 
production  representative  configuration,  which  includes  a  different  laptop  computer. 
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